Relationship mapping employing multi-dimensional context including facial recognition

ABSTRACT

A system and method for mapping interpersonal relationships, the method including processing a multiplicity of images and contextual information relating thereto including creating and prioritizing a list of a plurality of candidate persons having at least a predetermined relationship with at least one person connected to at least one image, using multi-dimensional information including visually sensible information in the multiplicity of images and contextual information relating thereto and searching the list of a plurality of candidate persons based at least in part on the prioritizing to select at least one of the candidate persons as having at least a predetermined relationship with the at least one person.

PRIORITY REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. 120 of U.S. patentapplication Ser. No. 12/922,984, filed 15 Feb. 2011, which claims thebenefit under 35 U.S.C. §365(c) of International Patent Application No.PCT/IL2009/000316, filed 19 Mar. 2009, which claims the benefit under 35U.S.C. §119(e) of U.S. Provisional Patent Application No. 61/070,377,filed 20 Mar. 2008, each of which is incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to systems and methods for mappinginterpersonal relationships.

BACKGROUND OF THE INVENTION

The following publications are believed to represent the current stateof the art:

U.S. Pat. Nos. 5,164,992; 5,963,670; 6,292,575; 6,819,783; 6,944,319;6,990,217; 7,274,822 and 7,295,687; and

U.S. Published Patent Application Nos.: 2006/0253491 and 2007/0237355.

SUMMARY OF THE INVENTION

The present invention seeks to provide an improved system for mappinginterpersonal relationships. There is thus provided in accordance with apreferred embodiment of the present invention a method for mappinginterpersonal relationships including processing a multiplicity ofimages and contextual information relating thereto including creatingand prioritizing a list of a plurality of candidate persons having atleast a predetermined relationship with at least one person connected toat least one image, using multi-dimensional information includingvisually sensible information in the multiplicity of images andcontextual information relating thereto and searching the list of aplurality of candidate persons based at least in part on theprioritizing to select at least one of the candidate persons as havingat least a predetermined relationship with the at least one person.

There is additionally provided in accordance with a preferred embodimentof the present invention a system for mapping interpersonalrelationships including processing functionality operative to process amultiplicity of images and contextual information relating theretoincluding creating and prioritizing a list of a plurality of candidatepersons having at least a predetermined relationship with at least oneperson connected to at least one image, using multi-dimensionalinformation including visually sensible information in the multiplicityof images and contextual information relating thereto and searchingfunctionality operative to search the list of a plurality of candidatepersons based at least in part on the prioritizing to select at leastone of the candidate persons as having at least a predeterminedrelationship with the at least one person.

Preferably, the searching also employs the multi-dimensionalinformation.

In accordance with a preferred embodiment of the present invention, themulti-dimensional information includes at least one of visualinformation appearing in an image, geographical information appearing inthe image, visual background information appearing in the image, imagesof other persons appearing in the image and person identifiers appearingin the image and at least one of information relating to an imagecollection of which the image forms a part, a time stamp associated withthe image, information not appearing in the image but associatedtherewith, geographical information not appearing in the image, visualbackground information appearing in another image and person identifiersnot appearing in the image but otherwise associated therewith.

More preferably, the multi-dimensional information includes at least oneof visual information appearing in an image, geographical informationappearing in the image, visual background information appearing in theimage, images of other persons appearing in the image and personidentifiers appearing in the image and at least two of informationrelating to an image collection of which the image forms a part, a timestamp associated with the image, information not appearing in the imagebut associated therewith, geographical information not appearing in theimage, visual background information appearing in another image andperson identifiers not appearing in the image but otherwise associatedtherewith.

Even more preferably, the multi-dimensional information includes atleast one of visual information appearing in an image, geographicalinformation appearing in the image, visual background informationappearing in the image, images of other persons appearing in the imageand person identifiers appearing in the image and at least three ofinformation relating to an image collection of which the image forms apart, a time stamp associated with the image, information not appearingin the image but associated therewith, geographical information notappearing in the image, visual background information appearing inanother image and person identifiers not appearing in the image butotherwise associated therewith.

Most preferably, the multi-dimensional information includes all of thefollowing: visual information appearing in an image, geographicalinformation appearing in the image, visual background informationappearing in the image, images of other persons appearing in the image;person identifiers appearing in the image, information relating to animage collection of which the image forms a part, a time stampassociated with the image, information not appearing in the image butassociated therewith, geographical information not appearing in theimage, visual background information appearing in another image andperson identifiers not appearing in the image but otherwise associatedtherewith.

In accordance with a preferred embodiment of the present invention, theat least one image is a composite image, which includes images ofmultiple persons, at least one of whom is unknown. Alternatively, the atleast one image is a composite image, which includes images of multiplepersons, none of whom are known.

Preferably, the at least one person connected with the at least oneimage appears in the image.

In accordance with a preferred embodiment of the present invention, themethod also includes tagging the at least one person connected with theat least one image with a person identifier. Additionally oralternatively, the processing includes iterative generation ofrelationship maps based on at least visually sensible information andalso on additional, non-visually sensible information related to personswho either appear in the at least one image or are otherwise associatedtherewith.

Preferably, the non-visually sensible information is meta-dataassociated with image data. Additionally, the meta-data includes dataderived from a social network. Alternatively or additionally, themeta-data includes data attached to the image data of the compositeimage.

In accordance with a preferred embodiment of the present invention, theiterative generation of relationship maps starts from a precursorrelationship map, containing information on relationships of at leastone known person in the at least one image.

Preferably, the relationship map is also based on inter-personalrelationship data received from at least one of a social network andearlier instances of relationship mapping based on analysis of otherimages. Additionally or alternatively, the relationship map includes anindication of at least strength of the relationship between two persons.Alternatively or additionally, the relationship map includes a facerepresentation which identifies each of the persons in the map.

In accordance with a preferred embodiment of the present invention, therelationship map includes an indication of whether each person in themap is a male or female. Additionally, the indication of whether eachperson in the map is a male or female is provided by at least one of asocial network and operation of image attribute recognition.

Preferably, the prioritizing employs an indication of whether a personis a male or female. Additionally or alternatively, the prioritizingemploys an indication of whether a person appears in the same album in asocial network as another person appears.

In accordance with a preferred embodiment of the present invention, theprocessing includes seeking candidate persons having at least apredetermined relationship with a known person in at least one image.Additionally, the seeking candidate persons is carried out by startingwith the generation of a list of candidate persons who have a temporalassociation with the known person based on visually-sensible informationcontained in the at least one image as well as non-visually sensibleinformation. Additionally, the non-visually sensible informationincludes at least one of the time and geographical location where thecomposite image was taken and an identification of an album on a socialnetwork with which it is associated. Additionally or alternatively, thenon-visually sensible information is obtained at least by interfacingwith social network APIs to find persons who appeared in other picturesin the same album, or persons that appeared in other albums taken in thesame geographical location at the same time.

Preferably, the list of candidate persons is extended and furtherprioritized by analyzing relationships of the persons appearing in arelationship map. In accordance with a preferred embodiment of thepresent invention, the prioritizing employs image attribute filtering.

In accordance with a preferred embodiment of the present invention, theprocessing includes facial representation generation performed on anunknown person in at least one image. Additionally, the method alsoincludes comparing the facial representation with previously generatedfacial representations of the candidate persons in accordance with andin an order established by the prioritizing. Preferably, the comparingis terminated and a candidate person is selected when a combinedpriority/similarity threshold is reached for a given candidate person,the priority/similarity threshold taking into account the similarity ofa facial representation of a candidate person to the facialrepresentation of an unknown person, the priority of that candidateperson established by the above-referenced prioritizing and the qualityof the facial representation of the candidate person.

Preferably, user feedback confirming that the person whose image isbelieved to be a given person is or is not that person is employed ingeneration of a further iterative relationship map. Alternatively, userfeedback confirming that the person whose image is believed to be agiven person is or is not that person is employed in improving the facerepresentation.

In accordance with a preferred embodiment of the present invention, thesearching includes comparison of face representations of persons in thelist of candidate persons with face representations of persons in therelationship map.

In accordance with a preferred embodiment of the present invention, themethod also includes searching at least one of the relationship maps.Additionally, the searching of the at least one of the relationship mapsemploys search terms including at least one of uniquely identifiedpersons, an additional image of a person, relationships between variouspersons, gender and face representations. Alternatively or additionally,the searching of the at least one of the relationship maps employs auser interface. Additionally or alternatively, the searching of the atleast one of the relationship maps is carried out via at least onesocial network having access to the at least one of the relationshipmaps.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be more fully understood and appreciated fromthe following detailed description, taken in conjunction with thedrawings in which:

FIGS. 1A and 1B are together a simplified generalized illustration ofrelationship mapping functionality, employing multi-dimensional context,including facial recognition, operative in accordance with a preferredembodiment of the present invention; and

FIG. 2 is a simplified functional block diagram of a system forrelationship mapping employing multi-dimensional context includingfacial recognition in accordance with a preferred embodiment of thepresent invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Reference is now made to FIG. 1A, which is a simplified generalizedillustration of relationship mapping functionality employingmulti-dimensional context, including facial recognition, operative inaccordance with a preferred embodiment of the present invention. Facialrecognition preferably includes facial representation generation andsubsequent comparison of multiple facial representations.

The functionality may be understood and visualized by starting with acomposite image, represented by a line drawing 100, which includesimages of multiple people, at least one of whom is known. In the presentexample, exemplified by line drawing 100, one person, here labeled John,is known and a second person, here labeled Unknown, is not known. Inthis example, the person who took the picture represented by linedrawing 100 is also known.

In accordance with a preferred embodiment of the invention in order toidentify an unknown person, an iterative relationship map is generatedbased inter alia on visually sensible information contained in thecomposite image represented by line drawing 100 and also on additional,non-visually sensible information related to the above-mentioned personswho either appear in the composite image or are otherwise associatedtherewith. In a preferred embodiment, the non-visually sensibleinformation may be meta-data attached to or associated with image data.The image data typically includes images in JPEG or PNG format. Themeta-data may be data in XML or other suitable formats derived from asocial network, such as FACEBOOK®, MYSPACE® AND FLICKR® as well as dataconventionally attached to image data, such as XML, EXIF tag or otherstandard image contextual data. Typically, in the present example, Johnand Peter are uniquely known on a social network. The person who tookthe picture containing the composite image represented by line drawing100 is identified as Greg, preferably by XML data attached to the imagedata of the composite image represented by line drawing 100.

Generation of the relationship map preferably starts from a pre-existingiterative relationship map, here termed a precursor relationship map,represented by a diagram 102, containing information on relationships ofa known person or known persons in the composite image, in this caseJohn. The precursor relationship map is also based on the inter-personalrelationship data received from one or more social networks as well asinter-personal relationship data derived from earlier instances ofoperation of the relationship mapping functionality of the presentinvention based on analysis of other composite images.

Diagram 102 indicates that John, a male, is known to have a socialrelationship with Sharon, a female, who in turn has a socialrelationship with Mike, a male. John also has a social relationship withPeter, a male. The symbology employed in the relationship map indicatesvarious parameters, including strength of the relationship between twopersons. In the present example, a number inserted in a relationshipindicating arrow indicates the strength of the relationship in thedirection indicated by the arrow. The higher the number, the strongerthe relationship in the illustrated example.

In the example of diagram 102, the relationship between John and Mike isexpected to be relatively strong, by virtue of the relationship betweenJohn and Sharon (85) and the relationship between Sharon and Mike (100),notwithstanding that it is an indirect relationship, through Sharon.This strength may be evidenced by multiple composite images in whichSharon appears with Mike and separately with John. The relationshipbetween John and Peter is indicated, by the number 10, to be relativelyweak, notwithstanding that it is a direct relationship. For example Johnand Peter may both appear together only in one composite image and thatimage may include many other people.

The precursor relationship map also includes a face representationproduced by conventional facial representation generation techniques,such as techniques described in either or both of U.S. Pat. No.5,164,992, entitled “Face recognition system” and U.S. Pat. No.6,292,575, entitled “Real-time facial recognition and verificationsystem”. The face representation is typically in the form of a vector,which identifies each of the persons in the map.

The precursor relationship map also includes an indication of whethereach person in the map is a male or female, indicated by the letters Mand F. This indication may be provided by various sources, such as asocial network or by operation of image attribute recognition, which maybe entirely conventional, such as described in U.S. Pat. No. 6,990,217entitled: “Gender classification with support vector machines”.Additional attributes may be generated by image attribute recognitionand can be included within the precursor relationship map. These mayinclude complexion, eye color and hair color. Conventional imageattribute recognition is known to have accuracy of above 90% indetermining gender.

The precursor relationship map and subsequent relationship mapspreferably also include information from visual background analysis.

Generation of the relationship map employs information from thecomposite image represented by line drawing 100, such as that Johnappears in the composite image together with an unknown individual.Image attribute analysis is preferably applied to the composite imagerepresented by line drawing 100, in order to determine whether theunknown individual is a male or a female.

In accordance with a preferred embodiment of the present invention,candidate persons having at least a predetermined relationship with theknown person, John, in the composite image are sought. This ispreferably done by starting with the generation of a list of candidatepersons who have a temporal association with the known person based onvisually-sensible information contained in the composite image as wellas the non-visually sensible information typically available asmeta-data.

Such non-visually sensible information may include the time andgeographical location where a picture was taken and the album on asocial network with which it is associated. For example, by interfacingwith social network APIs, queries can be made to find persons whoappeared in other pictures in the same album, or persons that appearedin other albums taken in the same geographical location at the sametime. These persons would typically be on an initial list of candidatepersons.

In accordance with a preferred embodiment of the present invention, thelist of candidate persons is extended and further prioritized byanalyzing relationships of the persons appearing in the precursorrelationship map. In practice, the precursor relationship map mayinclude millions of people. It is a particular feature of the presentinvention that prioritization of the persons appearing in the precursorrelationship map is carried out. This prioritization preferably includesimage attribute filtering, which eliminates persons who are of a genderother than the gender of the unknown person in the composite image. Forexample, referring to diagram 102, the persons appearing are Mike andSharon. Image attribute filtering is used to eliminate Sharon, since,image attribute recognition indicates that the unknown person in thecomposite image represented by line drawing 100 is a male.

The prioritization preferably relies heavily on the strengths ofrelationships between the known person and various other persons in theprecursor relationship map and gives much higher priority to personshaving the strongest relationship with the known person. Thus in thepresent example, Mike is prioritized over Peter. The prioritization isgiven expression in operation of the functionality of the presentinvention preferably by initially performing facial recognition on theimages of persons having highest priority. Thus, when the pool ofcandidates includes millions of people, the prioritization which is aparticular feature of the present invention, is of great importance.

Facial representation generation, which may be entirely conventional, isperformed on the unknown person in the composite image represented byline drawing 100. The resulting facial representation is compared withpreviously generated facial representations of the candidate persons inaccordance with and in the order established by the above-describedprioritization. The result of the comparison is a metric depicting thesimilarity between the two different facial representations. Thecomparison is cut off and a candidate is selected when a combinedpriority/similarity threshold is reached for a given candidate person.

The priority/similarity threshold takes into account the similarity of afacial representation of a candidate person to the facial representationof the unknown person, the priority of that candidate person establishedby the above-referenced prioritization and a metric which indicates thequality of the facial representation of the candidate person. Thismetric is a function of various parameters, such as the number of imagesof that person that have been analyzed by the system and previous userfeedback. A preferred quality metric, Qi, is given by the followingexpression:

${Qi} = {\left\lbrack {\left\lbrack {1 - \left( \frac{1}{n} \right)^{2}} \right\rbrack \times q} \right\rbrack \times \left\lbrack {\frac{tp}{fp} \times \left( \frac{1}{fn} \right)^{2}} \right\rbrack}$

where n is the count of images including the face representation, fp isthe percent of false positives indicated so far by user feedback, tp isthe percent of true positives indicated so far by user feedback, fn isthe percent of false negatives indicated so far by user feedback and qis a weighting of variance of the vectors representing the images thatcompose the face representation.

The match between the unknown person and the selected candidate personis then employed to provide an initial revised relationship map,indicated by a diagram 104. In the illustrated example, the unknownperson is tentatively identified as Mike and the relationship betweenMike and John is initially indicated as being a relatively weakrelationship. It is noted that Greg also appears in diagram 104 ashaving a weak one-directional relationship with John, based on Greghaving taken the photograph containing the composite image representedby line drawing 100.

If any positive user feedback is received via a social networkconfirming that the person whose image is believed to be Mike is indeedMike, this feedback is used to strengthen the relationship between Mikeand John as expressed in a subsequent revised relationship map, notshown, and to strengthen the metric which indicates the quality of thefacial representation of Mike. Conversely, receipt of negative feedbackindicating that the person whose image is believed to be Mike is notMike weakens the relationship between Mike and John as expressed in asubsequent revised relationship map, not shown, and weakens the metricwhich indicates the quality of the facial representation of Mike.Additionally it serves as a negative example for future facialrepresentation comparison.

Reference is now made to FIG. 1B, which is another simplifiedgeneralized illustration of relationship mapping functionality employingmulti-dimensional context, including facial recognition, operative inaccordance with a preferred embodiment of the present invention.

The functionality may be understood and visualized by starting with acomposite image, represented by a line drawing 200, which includesimages of multiple people. In the present example, exemplified by linedrawing 200, three persons here labeled Unknown 1, Unknown 2 and Unknown3, appear. All are not known. In this example, the person who uploadedthe picture to the social network site represented by line drawing 200is known to be the abovementioned John. Identification of the unknownpersons preferably employs relationship mapping.

Generation of a relationship map preferably begins from a pre-existingiterative relationship map, for example a precursor relationship map,represented by a diagram 202, which is identical to diagram 104. Thisprecursor relationship map contains information on relationships of aknown person or known persons in the previously analyzed compositeimage, in this case John, Peter, Greg and Sharon. This information isbased on the inter-personal relationship data received from one or moresocial networks as well as inter-personal relationship data derived fromthe earlier instance of operation of the relationship mappingfunctionality of the present invention based on analysis of othercomposite images.

Diagram 202 indicates that John, a male, is known to have a strongsocial relationship with Sharon, a female, who in turn has a strongsocial relationship with Mike, a male. John is also indicated to haveweak social relationships with Peter, Greg and Mike, who are males.

In accordance with a preferred embodiment of the present invention,candidate persons having at least a predetermined relationship with theknown person, John, who uploaded the picture represented by line drawing200, are sought. This is preferably done by starting with the personsappearing in the precursor relationship map 202. As noted above, it is aparticular feature of the present invention that prioritization of thepersons appearing in the precursor relationship map is carried out.

The prioritization preferably relies heavily on the strength of therelationship between the known person, John, and other persons in therelationship map 202 and gives much higher priority to persons havingthe strongest relationship with the known person, John. Thus in thepresent example, John is prioritized above all, as having the strongestrelationship to himself. After John, Mike has the next highest priority,since Sharon is eliminated by her gender.

After Mike, Peter has a higher priority than Greg, notwithstanding thatboth of their relationship arrows are given the same numerical score,since the relationship between John and Greg is only known to beunidirectional.

Prioritization preferably is also based on a certainty metric. In thiscase, the certainty that John is one of the unknown persons in thecomposite image 200 initially is not particularly high. In view of this,a prioritization cut-off is implemented, such that Peter and Greg, whohave relatively weak relationships with John, are not considered to becandidates. As noted above, prioritization is given expression inoperation of the functionality of the present invention preferably byinitially performing facial recognition on the persons having highestpriority, starting with John.

Facial representation generation is performed on the unknown persons inthe composite image represented by line drawing 200. The resultingfacial representation is compared with previously generated facialrepresentations of the candidate persons in accordance with and in theorder established by the above-described prioritization.

For example, facial representation generation is performed on the threeunknown images within composite image represented by a line drawing 200.Thereafter comparison of the facial representations of the three unknownpersons is carried out in accordance with the prioritized list generatedabove. The priority/similarity threshold for each is evaluated, and thusUnknown 1 is recognized as John, Unknown 2 and Unknown 3 are yet to berecognized.

In accordance with a preferred embodiment of the present invention,following recognition of Unknown 1 as John, in order to recognize theremaining unknown persons in the composite image, an additionalprioritization iteration is carried out. In this additionalprioritization iteration, the identification of Unknown 1 as Johnincreases the certainty metric for persons known to have a relationshipwith John and thus Peter is considered to be a candidate. Greg is stilltypically not considered to be a candidate since his relationship withJohn is unidirectional. Mike is typically not considered again inasmuchas a previous comparison of Mike with the generated unknown facerepresentation generated a low similarity metric.

A new priority list includes Peter, based on his relationship with John,who is now known to be previously Unknown 1 in the composite imagerepresented by line drawing 200.

Facial representations of the remaining unknown persons in the compositeimage represented by line drawing 200 are compared with previouslygenerated facial representations of the candidate persons in accordancewith and in the order established by the revised prioritization.

For example, Unknown 2 is recognized as Peter and Unknown 3 is yet to berecognized.

In accordance with a preferred embodiment of the present invention,following recognition of Unknown 2 as Peter, in order to recognize thelast unknown person in the composite image, a further prioritizationiteration is carried out. In this further prioritization iteration, theidentification of Unknown 2 as Peter indicates that there are twostarting points for generation, of candidate lists, John and Peter, bothof whom are known to be in the composite image. Two candidate listsubsets may thus be provided and used to generate a single prioritizedlist by using weighted graph combination techniques, as known in theart.

At this stage a further relationship map is generated, as indicated byreference numeral 204. In this relationship map, the indicatedrelationship between John and Peter is strengthened. Relationshipsbetween Unknown 3, John and Peter are also indicated based on thecomposite image represented by line drawing 200.

Unknown 3 may then be recognized in the future by comparing the facialrepresentation of Unknown 3 with facial representations of persons whoare subsequently indicated to have relationships with John or with theother persons appear in the relationship map 204.

Reference is now made to FIG. 2, which is simplified functional blockdiagram of a system for relationship mapping employing multi-dimensionalcontext including facial recognition in accordance with a preferredembodiment of the present invention. As seen in FIG. 2, the presentinvention utilizes one or more publicly available social networkapplication program interfaces (APIs) 300, such as the APIs provided byFACEBOOK®, MYSPACE® AND FLICKR®. Examples of such APIs include theFacebook API, Facebook Connect, Picasa Web Albums Data API and theFlickr Services API.

The system communicates interactively with the APIs 300 via widgets 302which may be embedded within applications such as FACEBOOK®, MYSPACE®AND FLICKR®, or standalone applications such as local album indexers304. The system automatically receives updates from APIs 300 viacrawlers 306, such as image crawlers, video crawlers and relationshipcrawlers, such as those used by spammers. Elements 302, 304 and 306preferably include user interface functionality. The user interfacefunctionality may be used to provide positive or negative feedbackregarding whether a recognized person is indeed the named person. Thisfeedback is communicated to relationship mapping coordinationfunctionality 310 and used to strengthen or weaken the facerepresentation. Additional user interface functionality includes searchfunctionality operative to search the generated relationship map. Searchterms include, for example, uniquely identified persons, an additionalimage of a person, relationships between various persons, other systemgenerated attributes such as gender or facial representation resemblanceand any suitable combination of the above. Search functionality can beprovided directly via a user interface or indirectly by exposing therelationship mapper 322 information to social networks.

Standalone applications may include running on an end-user machine andperforming some or all of the image attribute analysis, facialrepresentation generation and facial representation comparison. In apreferred embodiment of the present invention, a local album indexer 304performs image attribute analysis, facial representation generation andfacial representation comparison operations, and communicates with therelationship mapping coordination functionality 310 to generate aunified facial representation from multiple images of a single person.

Relationship mapping coordination functionality is preferably responsiveboth to API sourced information from APIs 300 and to user inputsreceived via communicators such as widgets 302, local album indexers 304and crawlers 306 and coordinates operation of the various elements ofthe system which are described hereinbelow.

The heart of the system of the present invention preferably includes anexpectation engine 320 which interfaces with a relationship mapper 322,which in turn interfaces with a relationship map database 324. Theseelements utilize information obtained by functionality 310 from facerecognition functionality 326 and attribute analysis functionality 328via an image analysis engine 330. Preferably a video analysis engine 332cooperates with interframe analysis functionality 334 and intraframeanalysis functionality 336, which provide information based on temporalsequences of frames in video content.

Relationship mapper 322 functionality preferably include providingaccess to a weighted graph of strengths of the relationships betweenvarious uniquely identified persons. Each node in the graph represents asingle person and includes a list of unique identifiers that can be oneor more of an email address, an internet identifier like OpenID or alist of IDs of specific social networks. In addition, the nodepreferably contains a facial representation generated from one or moreimages of the person, his or her gender and other information relatingto the person. The relationship map is stored at least partially inmemory and is preferably available persistently via relationshipdatabase 324.

The expectation engine 320 preferably generates prioritized lists ofcandidate persons, listing persons expected to appear in a compositeimage, its associated data and social network data. Initially, theexpectation engine queries the social networks via APIs 300 for a listof candidate persons having a temporal association with the known personbased on visually-sensible information contained in the composite imageas well as the non-visually sensible information typically available asmeta-data.

Subsequently, the expectation engine functionality performsprioritization of the candidate persons expected to appear in thecomposite image by interfacing with relationship mapper 322 and byutilizing image attribute filtering provided by the image analysisengine 330. The prioritization preferably relies heavily on the strengthof relationship between the known person and other persons in therelationship map and gives much higher priority to persons having thestrongest relationship with the known person. In a preferred embodiment,the expectation engine combines the weighted graphs associated withknown persons in the composite image, as provided by relationship mapper322 by utilizing weighted graph combination algorithms.

It will be appreciated by persons skilled in the art that the presentinvention is not limited to what has been particularly shown anddescribed hereinabove. Rather, the scope of the invention includes bothcombinations and sub-combinations of various features describedhereinabove as well as modifications and variations thereof which wouldoccur to a person skilled in the art upon reading the foregoingdescription and which are not in the prior art.

The invention claimed is:
 1. A method comprising: receiving, from aclient device of a first person associated with a social network, afirst image, the first image being a composite image that includes animage of at least one unknown person; processing a multiplicity ofimages and contextual information relating thereto including creatingand prioritizing a list of a plurality of candidate persons associatedwith the social network, the plurality of candidate personscorresponding to a plurality of person identifiers, respectively, eachcandidate person having at least a predetermined relationship with oneor more persons connected to the first image, the processing being basedon multi-dimensional information including facial-recognition processingof visually sensible information appearing in portions of saidmultiplicity of images and contextual information relating thereto;searching the list of the plurality of candidate persons based at leastin part on the prioritizing to select one of the candidate persons fortagging the unknown person in the first image, wherein the searchingcomprises comparing a face representation of the unknown person withface representations of each candidate person; and sending, to theclient device of the first user, the selected candidate person fortagging the image of the unknown person with the person identifiercorresponding to the selected candidate person.
 2. The method of claim1, wherein said searching also employs said multi-dimensionalinformation.
 3. The method of claim 1, wherein said multi-dimensionalinformation comprises: one or more of: visual information appearing inan image; geographical information appearing in said image; visualbackground information appearing in said image; images of other personsappearing in said image; or person identifiers appearing in said image;and one or more of: information relating to an image collection of whichsaid image forms a part; a time stamp associated with said image;information not appearing in said image but associated therewith;geographical information not appearing in said image; visual backgroundinformation appearing in another image; or person identifiers notappearing in said image but otherwise associated therewith.
 4. Themethod of claim 1, wherein the first image includes images of multiplepersons, none of whom are known.
 5. The method of claim 1, wherein theone or more persons connected to the first image appears in said image.6. The method of claim 1, further comprising tagging one or more of theone or more persons connected to the first image with a personidentifier.
 7. The method of claim 1, wherein said processing includesiterative generation of relationship maps based on at least visuallysensible information and also on additional, non-visually sensibleinformation related to persons who either appear in the first image orare otherwise associated therewith.
 8. The method of claim 7, whereinsaid iterative generation of relationship maps starts from a precursorrelationship map, containing information on relationships of at leastone known person in the first image.
 9. The method of claim 7, whereinsaid relationship map is also based on inter-personal relationship datareceived from at least one of a social network and earlier instances ofrelationship mapping based on analysis of other images.
 10. The methodof claim 7, wherein said relationship map includes an indication of atleast strength of the relationship between two persons.
 11. The methodof claim 7, wherein said relationship map includes a face representationwhich identifies each of the persons in the map.
 12. The method of claim7, wherein said relationship map includes an indication of whether eachperson in the map is a male or female.
 13. The method of claim 12,wherein said indication of whether each person in the map is a male orfemale is provided by at least one of a social network and operation ofimage attribute recognition.
 14. The method of claim 7, furthercomprising searching at least one of said relationship maps.
 15. Themethod of claim 14, wherein said searching of said at least one of saidrelationship maps employs search terms including at least one ofuniquely identified persons, an additional image of a person,relationships between various persons, gender and face representations.16. The method of claim 14, wherein said searching of said at least oneof said relationship maps employs a user interface.
 17. The method ofclaim 14, wherein said searching of said at least one of saidrelationship maps is carried out via at least one social network havingaccess to said at least one of said relationship maps.
 18. The method ofclaim 1, wherein said non-visually sensible information is meta-dataassociated with image data.
 19. The method of claim 18, wherein saidmeta-data includes data derived from a social network.
 20. The method ofclaim 18, wherein said meta-data includes data attached to the imagedata of said composite image.
 21. The method of claim 1, wherein saidprioritizing employs an indication of whether a person is a male orfemale.
 22. The method of claim 1, wherein said prioritizing employs anindication of whether a person appears in the same album in a socialnetwork as another person appears.
 23. The method of claim 1, whereinsaid processing includes seeking candidate persons having at least apredetermined relationship with a known person in at least one image.24. The method of claim 23, wherein said seeking candidate persons iscarried out by starting with the generation of a list of candidatepersons who have a temporal association with said known person based onvisually-sensible information contained in the first image as well asnon-visually sensible information.
 25. The method of claim 24, whereinsaid non-visually sensible information includes at least one of the timeand geographical location where said composite image was taken and anidentification of an album on a social network with which it isassociated.
 26. The method of claim 24, wherein said non-visuallysensible information is obtained at least by interfacing with socialnetwork APIs to find persons who appeared in other pictures in the samealbum, or persons that appeared in other albums taken in the samegeographical location at the same time.
 27. The method of claim 24,wherein said list of candidate persons is extended and furtherprioritized by analyzing relationships of the persons appearing in arelationship map.
 28. The method of claim 1, wherein said prioritizingemploys image attribute filtering.
 29. The method of claim 1, whereinsaid processing includes facial representation generation performed onan unknown person in at least one image.
 30. The method of claim 29,further comprising comparing said facial representation with previouslygenerated facial representations of the candidate persons in accordancewith and in an order established by said prioritizing.
 31. The method ofclaim 30, wherein said comparing is terminated and a candidate person isselected when a combined priority/similarity threshold is reached for agiven candidate person, said priority/similarity threshold taking intoaccount the similarity of a facial representation of a candidate personto the facial representation of an unknown person, the priority of thatcandidate person established by the above-referenced prioritizing andthe quality of the facial representation of the candidate person. 32.The method of claim 31, wherein user feedback confirming that the personwhose image is believed to be a given person is or is not that person isemployed in generation of a further iterative relationship map.
 33. Themethod of claim 31, wherein user feedback confirming that the personwhose image is believed to be a given person is or is not that person isemployed in improving said face representation.
 34. The method of claim1, wherein said searching includes comparison of face representations ofpersons in said list of candidate persons with face representations ofpersons in said relationship map.
 35. A system comprising: one or moreprocessors; and a memory coupled to the processors comprisinginstructions executable by the processors, the processors operable whenexecuting the instructions to: receive, from a client device of a firstperson associated with a social network, a first image, the first imagebeing a composite image that includes an image of at least one unknownperson; process a multiplicity of images and contextual informationrelating thereto including creating and prioritizing a list of aplurality of candidate persons associated with the social network, theplurality of candidate persons corresponding to a plurality of personidentifiers, respectively, each candidate person having at least apredetermined relationship with one or more persons connected to thefirst image, the processing being based on multi-dimensional informationincluding facial-recognition processing of visually sensible informationappearing in portions of said multiplicity of images and contextualinformation relating thereto; search the list of the plurality ofcandidate persons based at least in part on the prioritizing to selectone of the candidate persons for tagging the unknown person in the firstimage, wherein the searching comprises comparing a face representationof the unknown person with face representations of each candidateperson; and send, to the client device of the first user, the selectedcandidate person for tagging the image of the unknown person with theperson identifier corresponding to the selected candidate person. 36.One or more computer-readable non-transitory storage media embodyingsoftware that is operable when executed to: receive, from a clientdevice of a first person associated with a social network, a firstimage, the first image being a composite image that includes an image ofat least one unknown person; process a multiplicity of images andcontextual information relating thereto including creating andprioritizing a list of a plurality of candidate persons associated withthe social network, the plurality of candidate persons corresponding toa plurality of person identifiers, respectively, each candidate personhaving at least a predetermined relationship with one or more personsconnected to the first image, the processing being based onmulti-dimensional information including facial-recognition processing ofvisually sensible information appearing in portions of said multiplicityof images and contextual information relating thereto; search the listof the plurality of candidate persons based at least in part on theprioritizing to select one of the candidate persons for tagging theunknown person in the first image, wherein the searching comprisescomparing a face representation of the unknown person with facerepresentations of each candidate person; and send, to the client deviceof the first user, the selected candidate person for tagging the imageof the unknown person with the person identifier corresponding to theselected candidate person.